Method for Determining Severity of Skin Disease Based on Percentage of Body Surface Area Covered by Lesions

ABSTRACT

An image processing method is provided that automatically calculates Body Surface Area (BSA) score using machine learning techniques. A Felzenszwalb image segmentation algorithm is used to define proposed regions in each of a plurality of training set images. The training set images are oversegmented, and then each of the proposed regions in each of the plurality of oversegmented training set images are manually classified as being a lesion or a non-lesion. A Convolutional Neural Network (CNN) is then trained using the manually classified proposed regions in each of the plurality of training set images. The trained CNN is then used on test images to calculate BSA scores.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application Ser.No. 62/945,642, filed 9 Dec. 2019. The entire content of theaforementioned application is incorporated herein by reference in itsentirety.

BACKGROUND OF THE INVENTION

Disease severity evaluations of skin diseases such as Psoriasis involvescalculating the percentage of Body Surface Area that is covered bylesions and inflammation (i.e., BSA score). Hereafter, lesions andinflammation are collectively referred to as “lesions.” BSA is themeasured or calculated surface area of a human body.

Psoriasis is an autoimmune skin disease manifested as red andinflammatory areas that is distinct from healthy normal skin. Animportant part of disease severity measurements for Psoriasis is tomonitor what percentage of Body Surface Area is covered by inflamedareas called “lesions.” For Plaque Psoriasis, two major diseasemeasurements are BSA and PASI (Psoriasis Area and Severity Index), bothof which involve calculating a percentage score that is used to monitorthe disease progression and treatment effect (A Bozek, A. Reich (2017).The reliability of three psoriasis assessment tools: Psoriasis area andseverity index, body surface area and physician global assessment. AdvClin Exp Med. 2017 August; 26(5):851-856. doi: 10.17219/acem/69804).Currently these percentages are often estimated in physician's office bya dermatologist or nurse. A major problem with current Psoriasis diseasescores is that they are inexact and coarse estimations with humanbiases. Furthermore, the process to calculate the percentages to get anoverall PASI score is tedious and time consuming. Another clinical needis that there is currently no BSA measures for Guttate Psoriasis inwhich the body areas affected are large and covered by numerousinflammatory lesions ranging from 2 to 10 mm in size, making itdifficult to measure by eyesight.

Accordingly, there is a need for a more objective and quantitative wayto monitor the skin inflammation using computational methods. Thepresent invention fulfills such a need.

SUMMARY OF THE INVENTION

An image processing method is provided that automatically calculates BSAscore using machine learning techniques. A Felzenszwalb imagesegmentation algorithm is used to define proposed regions in each of aplurality of training set images. The training set images areoversegmented (“over segmented” or “over-segmented”), and then each ofthe proposed regions in each of the plurality of oversegmented trainingset images are manually classified as being a lesion or a non-lesion. AConvolutional Neural Network (CNN) is then trained using the manuallyclassified proposed regions in each of the plurality of training setimages. The trained CNN is then used on test images to calculate BSAscores. Also included in the present invention is a device (or computersystem) driven by computer instructions that is used in connection withthe method, e.g., computer-related device or medium for performing themethod as known in the relevant art.

BRIEF DESCRIPTION OF DRAWINGS

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawings will be provided by the Office upon request and paymentof the necessary fee.

The foregoing summary as well as the following detailed description ofpreferred embodiments of the invention, will be better understood whenread in conjunction with the appended drawings. For the purpose ofillustrating the invention, the drawings show presently preferredembodiments. However, the invention is not limited to the precisearrangements and instrumentalities shown. In the drawings:

FIG. 1 illustrates Psoriasis Image segmentation using Felzenszwalb,Quickshift, SLIC, and Compact watershed methods.

FIG. 2 illustrates a Psoriasis image correctly segmented by theFelzenszwalb method.

FIG. 3 illustrates three different types of inaccurately segmentedPsoriasis images: under-segmented, over-segmented and a whole-imagesegmented.

FIG. 4 illustrates how a CNN filter improves Felzenszwalb segmentationresults.

FIG. 5A illustrates under segmentation of an image.

FIG. 5B illustrates oversegmentation of an image.

FIG. 6 is a flowchart of a computerized method for determining severityof skin disease based on percentage of BSA that is covered by lesions,in accordance with one preferred embodiment of the present invention.

FIGS. 7A and 7B are schematic diagrams of system software and hardwarefor implementing FIG. 6 .

DETAILED DESCRIPTION OF THE INVENTION

Certain terminology is used herein for convenience only and is not to betaken as a limitation on the present invention.

I. OVERVIEW

FIG. 6 is a flowchart of a computerized method for determining severityof skin disease (e.g., psoriasis) based on percentage of BSA that iscovered by lesions. FIGS. 7A and 7B are schematic diagrams of systemsoftware and hardware for implementing FIG. 6 .

Referring to FIGS. 6, 7A, and 7B, the method operates as follows:

Step 600: Perform image segmentation on a plurality of training setimages of body surface areas using a Felzenszwalb segmentation algorithm(FSA), and output proposed regions in each of the plurality of trainingset images. Each of the plurality of training set images of body surfaceareas include skin disease. This image segmentation step is performed byprocessor 700 shown in FIG. 7A.Step 602: Oversegment each of the plurality of training set images. Thisstep is also performed by the processor 700.Step 604: Manually classify each of the proposed regions in each of theplurality of oversegmented training set images as being a lesion or anon-lesion. This step is performed by one or more human classifiers 702shown in FIG. 7A.Step 606: Train a neural network using the manually classified proposedregions in each of the plurality of training set images. This step isperformed by neural network 704 shown in FIG. 7A.Step 608: Perform image segmentation on a test image of a body surfacearea that includes skin disease using the Felzenszwalb segmentationalgorithm, and output regions in the test image of the body surfacearea. This image segmentation step is performed by processor 700′ shownin FIG. 7B. The processor 700′ may be the same, or different processoras the processor 700.Step 610: Oversegment the test image. This step is also performed by theprocessor 700′.Step 612: Input the regions of the oversegmented test image into thetrained neural network, labeled as 704′ in FIG. 7B because it is thesame neural network as neural network 704 in FIG. 7A, except that it isnow trained.Step 614: Use the trained neural network 704′ to identify and filter outnon-lesion regions from the oversegmented test image, wherein theremaining regions of the oversegmented test image are classified aslesion regions.Step 616: Calculate a percentage of BSA in the test image that iscovered by lesions using areas of the classified lesion regions of theoversegmented test image, and areas of the identified non-lesion regionsof the oversegmented test image. This step is performed in processor700″. The processor 700″ may be the same, or different processor as theprocessor 700 or processor 700′.

II. DETAILED DISCLOSURE

The detailed disclosure below describes the experimental process thatled to the present invention, and explains approaches that were moresuccessful than others.

The present invention treats the body surface area calculation as animage segmentation problem. Image segmentation has become one of thecornerstone issues within computer vision and is defined as the processof partitioning digital images into multiple segments, therebyorganizing image data into meaningful chunks. A series of imagesegmentation methods were investigated that can effectively calculatethe body surface area of a patient's psoriasis disease. Preferredembodiments of the presently claimed invention use the Felzenszwalbimage segmentation algorithm [3], and Convolutional Neural Networks (A.Krizhevsky, I. Sutskever, G. E. Hinton (2012) Imagenet classificationwith deep convolutional neural networks. Advances in Neural InformationProcessing (NeurIPS), 2012.) as a false positive filter to improve uponthe Felzenszwalb segmentation results.

Several image processing methods were tested to automatically calculateBSA score using machine learning techniques applied on 117 Guttate andPlaque Psoriasis images downloaded from the internet. In comparison toseveral other unsupervised segmentation methods, the Felzenszwalb imagesegmentation algorithm generated the highest percentages of correctsegmentations of lesion versus non-lesion areas with 56% of imageshaving a good segmentation while 44% of images are under or oversegmented. To improve the segmentation results, a Convolutional NeuralNetwork (CNN) was implemented, inspired by Visual Geometry Group (VGG)architecture to further filter out false positive lesions from theFelzenszwalb algorithm's proposed regions. Training data for the CNNwere comprised of human curated datasets of lesional and nonlesionalregions outputted from the Felzenszwalb algorithm. The CNN achieved a90% 5-fold cross validation testing accuracy score in classifyingbetween lesions and non-lesions. This CNN filter, applied alongside theFelzenszwalb algorithm, accurately segmented 77% of the training data or67 out of the 86 guttate psoriasis images. This method is useful fordigitizing disease severity measurements and remote monitoring of skindiseases such as Psoriasis for patients and physicians.

Image Data for Psoriasis

To gather the dataset, about 300 images of guttate psoriasis and 100images of chronic plaque psoriasis were scraped from Google Images.Inaccurate and/or misleading image data were filtered out, leaving afinal dataset of 86 guttate psoriasis images and 31 chronic plaquepsoriasis images.

Psoriasis Image Segmentation Methods

Five different image segmentation algorithms were tested, allimplemented using the scikit-learn and OpenCV library in Python. Thefive tested algorithms were Felzenszwalb, Quickshift, SLIC, Compactwatershed, and Otsu's Thresholding algorithm (see a review in D. Liu, B.Soran, G. Petrie, and L. Shapiro. A review of computer visionsegmentation algorithms. Lecture notes, 53, 2012.). After visuallyinspecting a couple image examples, it became apparent that theFelzenszwalb method (P. F. Felzenszwalb and D. P. Huttenlocher (2004)Efficient Graph-Based Image Segmentation. International Journal ofComputer Vision 59(2), 167-181.) produced the most promisingsegmentation by far.

FIG. 1 shows the output images of four different segmentationalgorithms. Yellow boundaries mark the segmentation of a suggested(proposed) region by the algorithm. After these tests, it was decided tosegment all images with the Felzenszwalb algorithm.

Psoriasis Segmentation and Binary Classification Filtering Algorithm

To enhance the Felzenszwalb segmentation results, a new algorithm wasused based upon a convolutional neural network filter that removes falsepositives from the segmentation. This algorithm is inspired by and holdssimilarities to the region-based convolutional neural networks (R-CNNs)that currently have become the state of the art in tackling imagesegmentation problems. The algorithm operates as follows:

1. Train a neural network to distinguish between lesions and non-lesionsamong proposed regions from the Felzenszwalb segmentation algorithm.

2. Oversegment the image by increasing the k-value parameter to anoptimal threshold (k=250).

3. Use the neural network to filter out the non-lesions proposed by theFelzenszwalb segmentation.

Oversegmenting the image in step 2 ensures that most lesions will beincluded inside the segmentation. Then, to filter out the excessnon-lesions or false positives the pre-trained neural network from step1 is used to distinguish non-lesion regions from lesion regions in step3.

Training Data

The proposed regions outputted from the Felzenszwalb segmentationalgorithm (with the k-value set to 250) were used to oversegment theimage, thereby including as many true positive lesion regions aspossible. Next, about 30 guttate psoriasis images were chosen, whichgenerated around 3000 proposed region images. Then, each of the 3000proposed regions were manually (human) classified as lesion ornon-lesion. To verify the accuracy of this dataset, the process wasrepeated three times. Some of these non-lesions were easy to identify,including huge regions covering a lot of skin, black background areas,necklaces, noise, and the like. Other regions were harder to distinguishbetween lesion and non-lesion including regions that had shade, badlighting, scars, and the like. A large source of error is believed to beattributed to the incongruencies in the dataset and these harder regionsto classify as lesion versus non-lesion. This may be another explanationfor why the binary classification results are still not close to thestate-of-the-art results seen in the Modified National Institute ofStandards and Technology (MNIST) or Canadian Institute for AdvancedResearch (CIFAR) datasets.

Binary Classification Neural Network Experiments

To train the neural network models a procedure was followed similar tothe one described below:

First, all proposed region inputs were preprocessed by resizing all ofthe images to a certain constant pixel size using the cubicinterpolation method from the opencv library. Next, different parametersof the neural network model were changed including model architecture(dense neural network vs convolutional neural network, hidden layersizes, batch normalization), parameters (learning rate), and input imageinterpolation pixel sizes (4×4, 8×8, 16×16, 32×32, 64×64). Finally, allof the models were tested with an 80-20 train-test data split ratio todetermine the accuracy, log-loss, and mean squared error scores of therespective models. This means that 80% of the data was used to build themodel and 20% of the remainder data (which was unseen by the model) wasused to evaluate the predictive strength of the model.

The final CNN model that was chosen was inspired by theVGG-architecture, did not use batch normalization, had smaller hiddenlayers (both in number of hidden layers and width of hidden layers), hada learning rate of 1e-4, and used the Adam optimizer. (Adam is anadaptive learning rate optimization algorithm that's been designedspecifically for training deep neural networks.) To ensure the accuracyof the final model, this model was tested with 5-fold cross validationand got an average 5-fold training accuracy score of 94% and an average5-fold testing accuracy score of 90%.

Things that Improved Accuracy Results

There were three approaches that greatly improved the binaryclassification accuracy of the neural network:

The first approach was to cubic interpolate the shape of the inputimages to a constant 16×16 pixel size. At first, when the images wereset to size 64×64, the classification accuracy hovered at extremely lowrates of 60%. As the size of images was lowered, it was discovered thata standard small dense neural network would achieve better and betterclassification performances until one reached an optimal 16×16 imagesize. A reason for this may be that most proposed regions were around16×16 sizes. Thus, interpolating more information may cause the regionto gain misleading information and interpolating less information maycause the region to lose valuable information.

The second approach was to use convolutional neural networks, as opposedto dense neural networks. It is well known that convolutional neuralnetworks perform better than dense neural networks in imageclassification tasks due to a myriad of reasons.

The third approach was that “less is more” when building convolutionalneural network model architectures. For instance, when the hidden layersizes were reduced, faster training times and more accurate testingvalidation scores were obtained. Specifically, reducing the first denselayer hidden size, which thereby reduces a significant number ofparameters, was particularly important. This may be due to the fact thatan excess of parameters may prevent the model from generalizing well andcause overfitting effects. This overfitting issue can best be seen in anear 10% accuracy loss between training and testing validation scores inlarge VGG-models.

Things that had No Effect or Worsened Segmentation Results

Several approaches did not improve accuracy results. More specifically,three approaches had negligible or negative effects on the model.

First, adding batch normalization layers as seen in the VGG models wasbelieved improve results, but as seen in certain papers (S. Santurkar,D. Tsipras, A. Ilyas, A. Madry (2018) How Does Batch Normalization HelpOptimization? Advances in Neural Information Processing (NeurIPS),2018.), batch normalization does not seem to improve classificationaccuracy in all cases.

Second, adjusting the learning rate did not provide performance gains.After testing the rates of 0.01, 0.001, 0.0003, and 0.0001 it was foundthat learning rate gives a negligible difference in performance resultsbetween 0.001, 0.0003, and 0.0001 learning rate parameters.

Third, deeper and wider network did not increase the accuracy of themodel. The larger dense neural network models and convolutional neuralnetwork models seemed to do worse by up to even 10% on cross validationtesting than the simple small VGG-models that was tried. This seems tosupport the founding principles of the agile developmentprinciple—simplicity.

Results and Discussion

i. Image Segmentation Algorithms Benefits and Drawbacks

After testing a small sample of images upon the five differentalgorithms, it became clear that the Felzenszwalb algorithm had farsuperior results than the other five unsupervised algorithms, as shownin FIG. 1 . An important characteristic of this method is its ability topreserve detail in low-variability image regions while ignoring detailin high-variability regions. It is also fast (<1 sec for 512×512 image)with runtime O(n log n) where n is number of pixels. Based upon theseobservations the Felzenszwalb method was chosen to be the main methodthat was applied for Psoriasis images.

ii. Felzenszwalb Image Segmentation Scoring Metric and Results

FIG. 2 shows an example of a good image segmentation. A good imagesegmentation is defined as any image segmentation that did not miss anyclearly significant lesional parts that a typical doctor should haveclassified.

FIG. 3 shows three examples of inaccurate image segmentations. The firstimage is an example of an under segmented region. In order to reducethis problem, one should decrease the k-value parameter. The secondimage is an example of an over segmented region. In order to alleviatethis issue, one should increase the k-value parameter. The last exampleis an instance of huge regions being classified as lesions. In order toalleviate this, one should increase the k-value parameter. FIGS. 5A and5B also show examples of under segmentation and oversegmentation.

The baseline Felzenszwalb algorithm was able to segment 49 out of 88input images with a good segmentation, 30 out of 88 input images wereunder segmented, and 9 out of 88 input images were over segmented. BSAscore calculations were generated at the end of each segmentationresult.

iii. Neural Network Based Filtering Results

25 different neural network models were tested with differentparameters, architects, and input sizes. Test accuracies of each modelswere calculated. Setting lower image sizes, choosing smaller VGG-basedConvolutional Neural Networks, and learning rates of 1e-4 produces thebest model with the highest test accuracy of 0.9.

iv. Convolutional Neural Network Filter and Felzenszwalb ImageSegmentation Results

After implementing Felzenszwalb image segmentation filtered by theconvolutional neural network, 67 out of 86 images provided goodsegmentation, 16 out of 86 images with an under segmentation, and 3 outof 86 images with an oversegmentation. Two input images were removedfrom the original Felzenszwalb image segmentation set due to inaccuraterepresentations of guttate psoriasis. An example of the convolutionalneural network improving segmentation results can be seen in the imagesof FIG. 4 .

In the image on the left of FIG. 4 , the convolutional neural networkwas able to filter out the large regions on the left side of the arm,all background white regions, and many small noise regions which weremanually classified as non-lesions. With better training data and moreextensive training, the neural network model can achieve even betterfiltering results and be used as an effective complement with theFelzenszwalb segmentation algorithm to remove false positivenon-lesions.

CONCLUSION

In retrospect, it was found that the Felzenszwalb image segmentationalgorithm produced a good baseline diagnosis algorithm that couldeffectively calculate the body surface area of lesions for a patient. Itwas also discovered that convolutional neural networks could classifywith great accuracy between proposed lesion and non-lesions from theFelzenszwalb segmentation outputs when given a good training dataset.Combining these two results, it was shown that the Felzenszwalb imagesegmentation algorithm combined with a convolutional neural networkfilter does a great baseline diagnosis by calculating the imagesegmentation and thereby body surface area score for the psoriasisdisease.

The BSA calculation methods described above may be used to generate adigitized Psoriasis disease score calculation system. For example, bytraining similar convolutional neural networks with 5 input images offront body, back, front leg, back leg, and head regions to automatingthe full PASI scoring system may output a severity index PASI scorebetween 0-72. Such computer systems, such as proposed recently in (C.Fink, L. Uhlmann, C. Klose, et al (2018) Automated, computer-guided PASImeasurements by digital image analysis versus conventional physicians'PASI calculations: study protocol for a comparative, single-centre,observational study BMJ Open 2018; 8: e018461. doi:10.1136/bmjopen-2017-018461.), can assist doctors in creating better,faster, and more informed decisions in diagnosis and monitoring of skindiseases such as Psoriasis.

It will be appreciated by those skilled in the art that changes could bemade to the embodiments described above without departing from the broadinventive concept thereof. It is understood, therefore, that thisinvention is not limited to the particular embodiments disclosed, but itis intended to cover modifications within the spirit and scope of thepresent invention.

1-6. (canceled)
 7. A system for determining severity of skin disease,comprising: a non-transitory computer-readable medium with instructionsstored thereon, which when executed by a processor perform stepscomprising: segmenting at least one test image into a set of imageregions; inputting the set of image regions to a trained neural network;with the trained neural network, classifying each region of the set ofimage regions as (a) lesion regions, (b) non-lesion body regions, or (c)non-body regions; calculating a percentage of body surface area (BSA)covered by lesions by using the combined areas of the lesion regions andthe combined areas of the lesion regions and non-lesion body regions. 8.The system of claim 7, wherein the trained neural network is aconvolutional neural network.
 9. The system of claim 7, wherein theinstructions further comprise the step of resizing each of the set ofimage regions to a constant pixel size and inputting the resized set ofimage regions to the trained neural network.
 10. The system of claim 9,wherein the constant pixel size is 16×16 pixels.
 11. The system of claim9, wherein the image regions are resized using bicubic interpolation.12. The system of claim 7, wherein the step of segmenting the testimages comprises using a Felzenszwalb image segmentation algorithm. 13.The system of claim 12, wherein the Felzenszwalb image segmentationalgorithm is executed with a k-value parameter of
 250. 14. The system ofclaim 7, wherein the step of segmenting the at least one test imagecomprises oversegmenting the at least one test image.
 15. The system ofclaim 7, wherein the skin disease is psoriasis.
 16. The system of claim7, wherein the instructions further comprise the step of calculating aPsoriasis Area and Severity Index (PAST) score from the calculatedpercentage of BSA covered by lesions
 17. A computerized method fordetermining severity of a skin disease in a subject, comprising:acquiring at least one test image of a subject; segmenting at least onetest image into a set of image regions; inputting the set of imageregions to a trained neural network; with the trained neural network,classifying each region of the set of image regions as (a) lesionregions, (b) non-lesion body regions, or (c) non-body regions;calculating a percentage of body surface area (BSA) covered by lesionsby using the combined areas of the lesion regions and the combined areasof the lesion regions and non-lesion body regions.
 18. The method ofclaim 17, wherein the trained neural network is a convolutional neuralnetwork.
 19. The method of claim 17, further comprising the step ofresizing each of the set of image regions to a constant pixel size andinputting the resized set of image regions to the trained neuralnetwork.
 20. The method of claim 19, wherein the constant pixel size is16×16 pixels.
 21. The method of claim 19, wherein the image regions areresized using bicubic interpolation.
 22. The method of claim 17, whereinthe step of segmenting the test images comprises using a Felzenszwalbimage segmentation algorithm.
 23. The method of claim 22, wherein theFelzenszwalb image segmentation algorithm is executed with a k-valueparameter of
 250. 24. The method of claim 17, wherein the step ofsegmenting the at least one test image comprises oversegmenting the atleast one test image.
 25. The method of claim 17, wherein the skindisease is psoriasis.
 26. The method of claim 17, further comprising thestep of calculating a Psoriasis Area and Severity Index (PAST) scorefrom the calculated percentage of BSA covered by lesions